In recent years, discrete cosine transformation (DCT) has been widely utilized in image coding processing. In MPEG as a representative image coding method, an input image signal is divided correspondingly to plural rectangular blocks constituting a single display screen as units of DCT processing, and DCT processing is performed block by block to the blocked image signal.
A specific description is given of image coding in MPEG.
FIG. 26 is a block diagram illustrating a construction of a conventional image processing apparatus which performs the above-mentioned image coding. In FIG. 26, reference numeral 200a designates a conventional image processing apparatus (image coding apparatus), which performs coding including DCT processing to an image signal. This image coding apparatus 200a consists of a blocking unit 102 for dividing an input image signal 101 correspondingly to plural blocks constituting a single display screen to generate an image signal (plural pixel values) 103 corresponding to each block, a DCT unit 104 for performing DCT processing to the image signal (pixel values) 103 to transform the image signal (pixel values) 103 into frequency components (DCT coefficients) 105, and a quantization unit 106 for quantizing the output 105 of the DCT unit 104 to generate quantized values 107 corresponding to each block. Herein, the DCT unit 104 and the quantization unit 106 constitute an information source coding unit 200a1.
Further, the image coding apparatus 200a consists of a scanner 109 for setting the processing order for coding the quantized values 107, and a variable-length coding unit (hereinafter referred to as a VLC unit) 112 for performing variable-length coding to quantized values 111 to which the processing order has been set, according to the set order, to generate a bit stream 113 corresponding to the image signal of each block.
A description is given of the operation.
Initially, the blocking unit 102 blocks an input image signal 101 correspondingly to rectangular blocks each comprising 8×8 pixels, and outputs an image signal (plural pixel values) 103 corresponding to each block. The DCT unit 104 transforms the image signal (pixel values) 103 into plural frequency components (DCT coefficients) 105 by DCT. The quantization unit 106 converts the DCT coefficients 105 into quantized values 107 by quantization.
Then, the scanner 109 performs rearrangement of the quantized values 107 so as to improve the efficiency of variable-length coding. That is, the scanner 109 sets the processing order for coding. Thereafter, the VLC unit 112 performs variable-length coding to the quantized values which have been rearranged, according to the set order. In addition, run length coding is used in variable-length coding processing. Therefore, when a scan is performed so that coefficients of about the same size are consecutive, the efficiency of variable-length coding is improved.
In coding an interlaced image signal, when correlations between adjacent scan lines are strong, frame DCT processing, i.e., DCT using a frame as a unit, is carried out. When correlations between scan lines in a field are strong, field DCT processing, i.e., DCT using a field as a unit, is carried out.
More specifically, as shown in FIG. 27, in frame DCT processing of an interlaced image signal, scan lines of a first field and scan lines of a second field are alternately arranged to form one frame-screen. This frame screen is divided into plural macroblocks each comprising 16×16 pixels. Each macroblock is divided into four subblocks-each comprising 8×8 pixels. Thereby, the image signal is subjected to DCT processing subblock by subblock. Meanwhile, in field DCT processing of an interlaced image signal, Each of macroblocks constituting one frame screen is formed by two first subblocks comprising only scan lines of a first field and two second subblocks comprising only scan lines of a second field. Thereby, the image signal is subjected to DCT processing subblock by subblock.
In MPEG, frame DCT or field DCT is adaptively selected for each macroblock. Accordingly, in order to perform accurate decoding to an input image signal, the blocking unit 102 in the image coding apparatus 200a outputs DCT processing information 114 indicating a unit of DCT processing for each macroblock (that is, information indicating whether each macroblock has been subjected to frame DCT or field DCT), together with the blocked image signal. Since a subblock which has been subjected to field DCT (a field DCT block) comprises only odd scan lines or only even scan lines among scan lines constituting one frame screen, a DCT coefficient group corresponding to the field DCT block includes more DCT coefficients indicating that the rate of change of pixel values in a vertical direction of a display screen is higher, as compared with a DCT coefficient group corresponding to a subblock which has been subjected to frame DCT (a field DCT block).
FIG. 28 is a block diagram illustrating a construction of an image decoding apparatus corresponding to the image coding apparatus shown in FIG. 26. In FIG. 28, reference numeral 200b designates an image processing apparatus (image decoding apparatus), which decodes the coded image signal 113 which has been coded by the image coding apparatus 200a. This image decoding apparatus 200b consists of a variable-length decoding unit (hereinafter referred to as a VLD unit) 201 for performing variable-length decoding to the coded image signal 113, and an inverse scanner 202 for performing an inverse scan to quantized values 111 which are obtained by decoding so that the order of the quantized values 111 is returned to the order before rearrangement in coding. Further, the image decoding apparatus 200b consists of an inverse quantization unit 203 for inverse-quantizing quantized values 107 which have been subjected to inverse scanning, to generate DCT coefficients (frequency components) 105 corresponding to a decoding target block to be subjected to decoding, an inverse DCT unit 204 for performing inverse DCT processing to the DCT coefficients 105 to generate an image signal (pixel values) 103 corresponding to the decoding target block, and an inverse blocking unit 205 for inverse-blocking the image signals 103 on the basis of the DCT processing information 114 from the image coding apparatus 200a, thereby regenerating an image signal 101 corresponding to one frame screen. Herein, the inverse quantization unit 203 and the inverse DCT unit 204 constitute an information source decoding unit 200b1.
In the image decoding apparatus 200b, inverse converting processes corresponding to the respective converting processes in the image coding apparatus 200a are carried out to a coded image signal, in the reverse order of the order in coding, thereby accurately decoding the coded image signal.
FIG. 29 is a block diagram illustrating a construction of another conventional image coding apparatus.
In FIG. 29, reference numeral 200c designated an image processing apparatus (image coding apparatus), which performs intra-frame predictive coding processing comprising generating predicted values of quantized values of a coding target block using information in a frame, and coding difference values between the predicted values and the quantized values of the coding target block.
This image coding apparatus 200c includes the image coding apparatus 200a, a prediction unit 200c2 for generating predicted values, and a scanning unit 200c1 for changing a scan method using a parameter concerning generation of the predicted values. The prediction unit 200c2 consists of a predictor 305 for generating predicted values 303, and outputting first prediction information 309a and second prediction information 309b concerning generation of the predicted values, an adder 301 for subtracting the output (predicted values) 303 of the predictor 305 from the output 107 of the quantization unit 106, and an adder 304 for adding the output 303 of the predictor 305 to an output 302 of the adder 301.
The scanning unit 200c1 consists of three scanners 109s1˜109s3 having different scan methods, for scanning the output 302 of the prediction unit 200c2, a first switch 108c for selecting one of the three scanners on the basis of a control signal 116 and supplying the output 302 of the prediction unit 200c2 to the selected scanner, a second switch 110c for selecting one of the three scanners on the basis of the control signal 116 and supplying an output of the selected scanner to the VLC unit 112, and a scan control unit 1401c for generating the control signal 116 on the basis of the first prediction information 309a. In addition, the second prediction information 309b is output from the image coding apparatus 200c.
In the image coding apparatus 200c thus constructed, a scan method is changed using the parameter concerning generation of predicted values (prediction information) 309, whereby the efficiency of variable-length coding is enhanced.
A description is given of a method for generating predicted values with reference to FIG. 30.
FIG. 30 shows a macroblock comprising 16×16 pixels. This macroblock comprises four subblocks (hereinafter simply referred to as blocks) R0, R1, R2 and X each comprising 8×8 pixels. The block X is a coding target block, and the blocks R0, R1 and R2 are already coded blocks which are adjacent to the coding target block X. Either block R1 or block R2 is referred in generating predicted values (quantized values) of the coding target block X. The block to be referred is decided using DC coefficients of the blocks R0, R1 and R2 (quantized values at the left upper ends of these blocks). Specifically, the absolute value of the difference between the DC coefficients of the blocks R0 and R1 is compared with the absolute value of the difference between the DC coefficients of the blocks R0 and R2. When the absolute value of the difference between the DC coefficients of the blocks R0 and R1 is larger, the block R1 is referred (reference in a vertical direction). When it is smaller, the block R2 is referred (reference in a horizontal direction).
When the block R1 is referred, the DC coefficient (the quantized value at the left upper end) of the block R1 and AC coefficients (quantized values at the uppermost line, except the DC coefficient) of the block R1 are used as predicted values of the coefficients of the block X at the same positions. When the block R2 is referred, the DC coefficient (the quantized value at the left upper end) of the block R2 and AC coefficients (quantized values at the leftmost line, except the DC coefficient) of the block R2 are used as predicted values of the coefficients of the block X at the same positions. In addition, in a case where the efficiency of variable-length coding is degraded by predicting AC coefficients, no AC prediction may be carried out.
A scan method is changed according to ON/OFF of Ac prediction (whether AC prediction is performed or not) in intra-frame prediction. Further, when AC prediction is in the ON state, a scan method is changed according to a reference direction of prediction. The first prediction information 309a supplied to the scan control unit 1401c includes ON/OFF information indicating ON/OFF of AC prediction, and prediction direction information indicating a reference direction for AC prediction, and the second prediction information 309b includes only the ON/OFF information of AC prediction.
When Ac prediction is in the OFF state, a scan of quantized values is executed in the order shown in FIG. 31(a). Thereby, the processing order for coding is set to the quantized values. In this case, in a group of quantized values corresponding to a subblock, high-frequency components uniformly distribute in vertical and horizontal directions very often. Therefore, the quantized values are uniformly scanned in the order from low-frequency components to high-frequency components. When AC prediction is performed and a vertical direction is referred, a scan of quantized values is executed in the order shown in FIG. 31(b). In this case, a group of quantized values corresponding to a subblock has a distribution in which high-frequency components in a horizontal direction are reduced by the prediction. Therefore, the quantized values are scanned with a priority given to a horizontal direction, thereby improving the efficiency of variable-length coding. When AC prediction is performed and a horizontal direction is referred, a scan of quantized values is executed in the order shown in FIG. 31(c). In this case, a group of quantized values corresponding to a subblock has a distribution in which high-frequency components in a vertical direction are reduced by the prediction. Therefore, the quantized values are scanned with a priority given to a vertical direction, thereby improving the efficiency of variable-length coding.
FIG. 32 is a block diagram illustrating a construction of an image decoding apparatus corresponding to the image coding apparatus shown in FIG. 29. In FIG. 32, reference numeral 200d designates an image processing apparatus (image decoding apparatus), which decodes the coded image signal 308 that has been coded in the image coding apparatus 200c.
This image decoding apparatus 200d has an inverse scanning unit 200d1 for performing an inverse scan to quantized values which are obtained by variable-length decoding of the coded image signal 308 so that the order of the quantized values is returned to the order before scanning in coding, and changing an inverse scan method on the basis of the prediction information (parameter) concerning generation of predicted values in the image coding apparatus 200c, and a prediction unit 200d2 for adding quantized values (predicted values) of a decoding target block which are predicted from quantized values of an already decoded block in the vicinity of the decoding target block, to the quantized values corresponding to the decoding target block which have been subjected to inverse scanning.
The inverse scanning unit 200d1 consists of three inverse scanners 202s1˜202s3 having different inverse scan methods, for inverse-scanning the output of the VLD unit 201, a first switch 108d for selecting one of the three inverse scanners on the basis of a control signal 116 and supplying the output of the VLD unit 201 to the selected inverse scanner, a second switch 110d for selecting one of the three inverse scanners on the basis of the control signal 116 and supplying the output of the selected inverse scanner to the prediction unit 200d2, and an inverse scan control unit 1401d for generating the control signal 116 on the basis of the first prediction information 309a.
In addition, the inverse scanner 202s1 performs an inverse scan corresponding to the scan by the scanner 109s1 in the image coding apparatus 200c, the inverse scanner 202s2 performs an inverse scan corresponding to the scan by the scanner 109s2 in the image coding apparatus 200c, and the inverse scanner 202s3 performs an inverse scan corresponding to the scan by the scanner 109s3 in the image coding apparatus 200c.
The prediction unit 200d2 consists of a predictor 401 for generating predicted values 303 on the basis of the second prediction information 309b output from the image coding apparatus 200c and values 107d corresponding to the quantized values 107 in the image coding apparatus 200c, and generating control prediction information 309a′ corresponding to the first prediction information 309a in the image coding apparatus 200c, and an adder 304 for adding the predicted values 303 to the output 302 of the inverse scanning unit 200d1. In addition, like the first prediction information 309a, the control prediction information 309a′ includes ON/OFF information of AC prediction and prediction direction information of AC prediction.
In the image decoding apparatus 200d thus constructed, inverse converting processes corresponding to the respective converting processes in the image coding apparatus 200c shown in FIG. 29 are carried out to a coded image signal, in the reverse order of the order in coding, thereby accurately decoding the coded image signal.
FIG. 33 is a block diagram illustrating a construction of still another conventional image coding apparatus. In FIG. 33, reference numeral 200e designates an image processing apparatus (image coding apparatus), which performs inter-frame predictive coding processing comprising generating predicted values of an image signal (pixel values) of a coding target frame from another frame, and coding difference values between the image signal (pixel values) of the coding target frame and the predicted values.
This image coding apparatus 200e has an information source coding unit 200e2 for performing information source coding to difference values 1002 between an image signal (pixel values) 103 obtained by blocking and predicted values 1008 of the image signal 103, in place of the information source coding unit 200a1 in the image coding apparatus 200a shown in FIG. 26, which performs information source coding to the image signal 103. Further, the image coding apparatus 200e has a scanning unit 200e1 for changing a scan method, i.e., the processing order for coding, according to a parameter 1015 concerning generation of the predicted values 1008, in place of the scanner 109 in the image coding apparatus 200a.
The information source coding unit 200e2 consists of an adder 1001, a DCT unit 104e, a quantization unit 106e, an inverse quantization unit 203e, an inverse DCT unit 204e, an adder 1010, a frame memory 1014, and a predictor 1012.
The adder 1001 is for subtracting predicted values 1008 from an image signal (pixel values) 103 corresponding to a coding target block. The DCT unit 104e is for transforming difference values 1002 between the image signal (pixel values) 103 and the predicted values 1008 into frequency components (DCT coefficients) 1003 by DCT. The quantization unit 106e is for quantizing the DCT coefficients 1003 to generate quantized values 1004 corresponding to the coding target block.
Further, the inverse quantization unit 203e is for inverse-quantizing the quantized values 1004 output from the quantization unit 106e to output DCT coefficients 1007 corresponding to the DCT coefficients 1003. The inverse DCT unit 204e is for performing inverse DCT to the DCT coefficients 1007 to output difference signals 1009 corresponding to the difference values 1002. The adder 1010 is for adding the predicted values 1008 to the difference signals 1009 to output an already coded image signal 1011 corresponding to the coding target block.
Furthermore, the frame memory 1014 is for temporarily storing already coded image signals 1011 corresponding to one frame or corresponding to frames of a prescribed number. The predictor 1012 is for generating the predicted values 1008 on the basis of an already coded image signal 1013 corresponding to a reference block in the memory 1014 and the image signal 103 corresponding to the coding target block.
The scanning unit 200e1 consists of two scanners 129s1 and 129s2 having different scan methods, for scanning the output of the information source coding unit 200e2, a first switch 108e for selecting one of the two scanners on the basis of a control signal 116e and supplying the output 1004 of the information source coding unit 200e2 to the selected scanner, a second switch 110e for selecting one of the two scanners on the basis of the control signal 116e and supplying an output of the selected scanner to the VLC unit 112, and a scan control unit 1016e for generating the control signal 116e on the basis of a parameter 1015 from the predictor 1012.
Herein, the scanner 129s1 performs a scan of quantized values in the order shown in FIG. 31(a). The scanner 129s2 is constituted by the respective elements 301, 304 and 305 in the prediction unit 200c2 shown in FIG. 29, and the respective elements 108c, 110c, 109s1˜109s3 and 1401c in the scanning unit 200c1 shown in FIG. 29. That is, the scanner 129s2 performs intra-frame prediction to a block to which no inter-frame prediction has been performed in coding (hereinafter referred to as an intra-coded block) and selects one of the scanners 109s1˜109s3 constituting the scanner 129s2 on the basis of prediction information concerning generation of predicted values. In addition, one of the scanners 109s1˜109s3 constituting the scanner 129s2 performs a scan of quantized values in the order shown in FIG. 31(a). The coding processing by the image coding apparatus 200e is fundamentally identical to that by the image coding apparatus 200c shown in FIG. 29, except that difference values between an image signal which is obtained by blocking and predicted values of the image signal are coded.
That is, in inter-frame predictive coding by the image coding apparatus 200e, predicted values 1008 are set to 0 when prediction efficiency is low, whereby an image signal 103 corresponding to a coding target block is subjected to DCT processing as it is (intra-coding). Switching between inter-coding and intra-coding is performed for each macroblock, and information indicating either inter-coding or intra-coding is added to a parameter 1015 concerning prediction.
Further, when a coding target block is an inter-coded macroblock, the scanner 129s1 is selected. When the coding target block is an intra-coded macroblock, the scanner 129s2 is selected. Thereby, a scan method suitable for each coding is executed.
Specifically, quantized values corresponding to an intra-coded macroblock are supplied to the scanner 129s2 comprising the prediction unit 200c2 and the scanning unit 200c1 shown in FIG. 29. In the scanner 129s2, predicted values of the quantized values are generated by intra-frame prediction, and an adaptive scan is performed to difference values between the quantized values of the coding target block and the predicted values, on the basis of prediction information concerning generation of the predicted values.
Meanwhile, quantized values corresponding to an inter-coded macroblock are supplied to the scanner 129s1, and a scan in the order shown in FIG. 31(a) is performed in the scanner 129s1.
In the image coding apparatus 200e thus constructed, since the difference values are coded, many DCT coefficients become 0 by quantization, whereby the efficiency of variable-length coding is improved.
In addition, in the image coding apparatus 200e, no intra-frame prediction may be carried out to an intra-coded macroblock. In this case, one of the scanners 109s1˜109s3 constituting the scanner 129s2 performs a scan in the order shown in FIG. 31(a) to quantized values of the intra-coded macroblock.
FIG. 34 is a block diagram illustrating a construction of an image decoding apparatus corresponding to the image coding apparatus 200e shown in FIG. 33. In FIG. 34, reference numeral 200f designates an image processing apparatus (image decoding apparatus), which decodes the coded image signal 1006 that has been coded in the image coding apparatus 200e.
This image decoding apparatus 200f has an inverse scanning unit 200f1 for performing an inverse scan to quantized values 1005 which are obtained by variable-length decoding of the coded image signal 1006 so that the order of the quantized values is returned to the order before scanning in coding, and changing an inverse scan method on the basis of the parameter 1015 concerning generation of predicted values in the image coding apparatus 200e, in place of the inverse scanner 202 in the image decoding apparatus 200b shown in FIG. 28. Further, the image decoding apparatus 200f has an information source decoding unit 200f2 for performing information source decoding to quantized values 1004 corresponding to a decoding target block which have been subjected to inverse scanning, in place of the information source de-coding unit 200b1 in the image decoding apparatus 200b.
The inverse scanning unit 200f1 consists of two inverse scanners 222s1 and 222s2 having different inverse scan methods, for inverse-scanning the output 1005 of the VLD unit 201, a first switch 108f for selecting one of the two inverse scanners on the basis of a control signal 116f and supplying the output 1005 of the VLD unit 201 to the selected inverse scanner, a second switch 110f for selecting one of the two inverse scanners on the basis of the control signal 116f and supplying the output of the selected inverse scanner to the information source decoding unit 200f2, and an inverse scan control unit 1016f for generating the control signal 116f on the basis of the prediction parameter 1015. Herein, the inverse scanners 222s1 and 222s2 correspond to the scanners 129s1 and 129s2 in the image coding apparatus 200e.
That is, the inverse scanner 222s1 performs an inverse scan corresponding to a scan in the order shown in FIG. 31(a), and the inverse scanner 222s2 is constituted by the respective elements 108d, 110d, 202s1˜202s3 and 1401d in the inverse scanning unit 200d1 shown in FIG. 32, and the respective elements 304 and 401 in the prediction unit 200d2 shown in FIG. 32.
The information source decoding unit 200f2 consists of an inverse quantization unit 203f for inverse-quantizing the output 1004 of the inverse scanning unit 200f1, an inverse DCT unit 204f for performing inverse DCT processing to an output 1003 of the inverse quantization unit 203f, an adder 1101f for adding predicted values 1008f of the decoding target block to an output 1002 of the inverse DCT unit 204f.
Further, the information source decoding unit 200f2 consists of a frame memory 1014f for temporarily storing already decoded image signals 103 corresponding to one frame or frames of a prescribed number, and a predictor 1102f for generating the predicted values 1008f of the decoding target block on the basis of an already decoded image signal 1013f corresponding to a reference block in the memory 1014f and the parameter 1015 concerning prediction in coding.
In the image decoding apparatus 200f thus constructed, inverse converting processes corresponding to the respective converting processes in the image coding apparatus 200e shown in FIG. 33 are carried out to a coded image signal, in the reverse order of the order in coding, thereby accurately decoding the coded image signal.
The scan changing method in any of the conventional image processing apparatuses is available for progressive image coding in which all blocks are frame DCT blocks. However, in interlaced image coding in which frame DCT blocks and field DCT blocks coexist, since a field DCT block and a frame DCT block have different distributions of DCT coefficients, coefficients of about the same size are not consecutive when the same scan changing method is used, so that the efficiency of variable-length coding is degraded.
That is, in interlaced image coding in which either frame DCT processing or field DCT processing is adaptively selected for each macroblock and macroblocks having different DCT types coexist, when a scan method is changed using a parameter concerning generation of predicted values, since a field DCT block and a frame DCT block have different distributions of DCT coefficients, coefficients of about the same size are not consecutive, so that the efficiency of variable-length coding is degraded.
Further, also in inter-frame predictive coding of an interlaced image in any of the conventional image processing apparatuses, the above-mentioned problem arises because macroblocks having different DCT types coexist.
Furthermore, also in coding of a progressive image, when switching is performed between frame DCT processing and field DCT processing according to the content of the image, for example, in a case where frame DCT processing is executed when correlations between adjacent scan lines are strong and field DCT processing is executed when correlations between adjacent scan lines are weak, the efficiency of variable-length coding is degraded as in the interlaced image coding.